Papers with Conceptual Captions dataset
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (P18-1)
Copied to clipboard
| Challenge: | Practical applications of automatic image description systems include leveraging descriptions for image indexing or retrieval, and helping those with visual impairments by transforming visual signals into information that can be communicated via text-to-speech technology. |
| Approach: | They propose to extract and filter image caption annotations from billions of webpages and use them to train models. |
| Outcome: | The proposed model architectures perform better when trained on the Conceptual Captions dataset. |
COSMic: A Coherence-Aware Generation Metric for Image Descriptions (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate captions have limited learning of their output . previous methods focused on n-gram measures of similarity to reference output based on a ngram of similarities to the output metric. |
| Approach: | They propose a first discourse-aware learned generation metric for evaluating image descriptions. |
| Outcome: | The proposed metric predicts human ratings of captions on out-of-domain images. |
Probing Image-Language Transformers for Verb Understanding (2021.findings-acl)
Copied to clipboard
| Challenge: | Multimodal image-language transformers have achieved impressive results on a variety of tasks that rely on fine-tuning. |
| Approach: | They collect a dataset of image-sentence pairs consisting of 421 verbs . they evaluate pretrained image-language transformers and find they fail more in situations that require verb understanding compared to other parts of speech. |
| Outcome: | The proposed model trains on a manually-annotated and smaller dataset does better on the task. |